{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning models as APIs using Flask\n",
    "\n",
    "### Introduction\n",
    "\n",
    "A lot of Machine Learning (ML) projects, amateur and professional, start with an aplomb. The early excitement with working on the dataset, answering the obvious & not so obvious questions & presenting the results are what everyone of us works for. There are compliments thrown around and talks about going to the next step -- that's when the question arises, __How?__\n",
    "\n",
    "The usual suspects are making dashboards and providing insights. But mostly, the real use of your Machine Learning model lies in being at the heart of a product -- that maybe a small component of an automated mailer system or a chatbot. These are the times when the barriers seem unsurmountable. Giving an example, majority of ML folks use `R/Python` for their experiments. But consumer of those ML models would be software engineers who use a completely different stack. There are two ways via which this problem can be solved:\n",
    "\n",
    "- __Rewriting the whole code in the language that the software engineering folks work__\n",
    "\n",
    "The above seems like a good idea, but the time & energy required to get those intricate models replicated would be utterly waste. Majority of languages like `JavaScript`, do not have great libraries to perform ML. One would be wise to stay away from it.\n",
    "\n",
    "- __API-first approach__\n",
    "\n",
    "Web APIs have made it easy for cross-language applications to work well. If a frontend developer needs to use your ML Model to create a ML powered web application, he would just need to get the `URL Endpoint` from where the API is being served. \n",
    "\n",
    "The articles below would help you to appreciate why APIs are a popular choice amongst developers:\n",
    "\n",
    "- [History of APIs](http://apievangelist.com/2012/12/20/history-of-apis/)\n",
    "- [Introduction to APIs - AV Article](https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-apis-application-programming-interfaces-5-apis-a-data-scientist-must-know/)\n",
    "\n",
    "Majority of the Big Cloud providers and smaller Machine Learning focussed companies provide ready-to-use APIs. They cater to the needs of developers/businesses that don't have expertise in ML, who want to implement ML in their processes or product suites.\n",
    "\n",
    "One such example of Web APIs offered is the [Google Vision API](https://cloud.google.com/vision/)\n",
    "\n",
    "![Google API Suite](http://www.publickey1.jp/2016/gcpnext16.jpg)\n",
    "\n",
    "All you need is a simple REST call to the API via SDKs (Software Development Kits) provided by Google. [Click here](https://github.com/GoogleCloudPlatform/cloud-vision/tree/master/python) to get an idea of what can be done using Google Vision API.\n",
    "\n",
    "Sounds marvellous right! In this article, we'll understand how to create our own Machine Learning API using `Flask`, a web framework with `Python`. \n",
    "\n",
    "![Flask](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Flask_logo.svg/640px-Flask_logo.svg.png)\n",
    "\n",
    "__NOTE:__ `Flask` isn't the only web-framework available. There's `Django`, `Falcon`, `Hug` and many more. For `R`, we have a package called [`plumber`](https://github.com/trestletech/plumber).\n",
    "\n",
    "### Table of Contents:\n",
    "\n",
    "1. __Python Environment Setup & Flask Basics__\n",
    "2. __Creating a Machine Learning Model__\n",
    "3. __Saving the Machine Learning Model: Serialization & Deserialization__\n",
    "4. __Creating an API using Flask__\n",
    "\n",
    "### 1. Python Environment Setup & Flask Basics\n",
    "\n",
    "![Anaconda](https://upload.wikimedia.org/wikipedia/en/c/cd/Anaconda_Logo.png)\n",
    "\n",
    "- Creating a virtual environment using `Anaconda`. If you need to create your workflows in Python and keep the dependencies separated out or share the environment settings, `Anaconda` distributions are a great option. \n",
    "    * You'll find a miniconda installation for Python [here](https://conda.io/miniconda.html)\n",
    "    * `wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh`\n",
    "    * `bash Miniconda3-latest-Linux-x86_64.sh`\n",
    "    * Follow the sequence of questions.\n",
    "    * `source .bashrc`\n",
    "    * If you run: `conda`, you should be able to get the list of commands & help.\n",
    "    * To create a new environment, run: `conda create --name <environment-name> python=3.6`\n",
    "    * Follow the steps & once done run: `source activate <environment-name>`\n",
    "    * Install the python packages you need, the two important are: `flask` & `gunicorn`.\n",
    "    \n",
    "    \n",
    "- We'll try out a simple `Flask` Hello-World application and serve it using `gunicorn`:\n",
    "\n",
    "    * Open up your favourite text editor and create `hello-world.py` file in a folder\n",
    "    * Write the code below:\n",
    "        ```python\n",
    "\n",
    "        \"\"\"Filename: hello-world.py\n",
    "        \"\"\"\n",
    "\n",
    "        from flask import Flask\n",
    "\n",
    "        app = Flask(__name__)\n",
    "\n",
    "        @app.route('/users/<string:username>')\n",
    "        def hello_world(username=None):\n",
    "\n",
    "            return(\"Hello {}!\".format(username))\n",
    "\n",
    "        ```\n",
    "    * Save the file and return to the terminal.\n",
    "    * To serve the API (to start running it), execute: `gunicorn --bind 0.0.0.0:8000 hello-world:app` on your terminal.\n",
    "    \n",
    "    * If you get the repsonses below, you are on the right track:\n",
    "\n",
    "    ![Hello World](https://raw.githubusercontent.com/pratos/flask_api/master/notebooks/images/flaskapp1.png)\n",
    "\n",
    "    * On you browser, try out: `https://localhost:8000/users/any-name`\n",
    "\n",
    "    ![Browser](https://raw.githubusercontent.com/pratos/flask_api/master/notebooks/images/flaskapp2.png)\n",
    "\n",
    "Viola! You wrote your first Flask application. As you have now experienced with a few simple steps, we were able to create web-endpoints that can be accessed locally. And it remains simple going forward too.\n",
    "\n",
    "Using `Flask`, we can wrap our Machine Learning models and serve them as Web APIs easily. Also, if we want to create more complex web applications (that includes JavaScript `*gasps*`) we just need a few modifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Creating a Machine Learning Model\n",
    "\n",
    "- We'll be taking up the Machine Learning competition: [Loan Prediction Competition](https://datahack.analyticsvidhya.com/contest/practice-problem-loan-prediction-iii). The main objective is to set a pre-processing pipeline and creating ML Models with goal towards making the ML Predictions easy while deployments. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os \n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Saving the datasets in a folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test.csv  training.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls /home/pratos/Side-Project/av_articles/flask_api/data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('../data/training.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Loan_ID',\n",
       " 'Gender',\n",
       " 'Married',\n",
       " 'Dependents',\n",
       " 'Education',\n",
       " 'Self_Employed',\n",
       " 'ApplicantIncome',\n",
       " 'CoapplicantIncome',\n",
       " 'LoanAmount',\n",
       " 'Loan_Amount_Term',\n",
       " 'Credit_History',\n",
       " 'Property_Area',\n",
       " 'Loan_Status']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(614, 13)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Finding out the `null/Nan` values in the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of null values in:Loan_ID == 0\n",
      "The number of null values in:Gender == 13\n",
      "The number of null values in:Married == 3\n",
      "The number of null values in:Dependents == 15\n",
      "The number of null values in:Education == 0\n",
      "The number of null values in:Self_Employed == 32\n",
      "The number of null values in:ApplicantIncome == 0\n",
      "The number of null values in:CoapplicantIncome == 0\n",
      "The number of null values in:LoanAmount == 22\n",
      "The number of null values in:Loan_Amount_Term == 14\n",
      "The number of null values in:Credit_History == 50\n",
      "The number of null values in:Property_Area == 0\n",
      "The number of null values in:Loan_Status == 0\n"
     ]
    }
   ],
   "source": [
    "for _ in data.columns:\n",
    "    print(\"The number of null values in:{} == {}\".format(_, data[_].isnull().sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Next step is creating `training` and `testing` datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pred_var = ['Gender','Married','Dependents','Education','Self_Employed','ApplicantIncome','CoapplicantIncome',\\\n",
    "            'LoanAmount','Loan_Amount_Term','Credit_History','Property_Area']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[pred_var], data['Loan_Status'], \\\n",
    "                                                    test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To make sure that the `pre-processing steps` are followed religiously even after we are done with experimenting and we do not miss them while predictions, we'll create a __custom pre-processing Scikit-learn `estimator`__.\n",
    "\n",
    "__To follow the process on how we ended up with this `estimator`, read up on [this notebook](https://github.com/pratos/flask_api/blob/master/notebooks/AnalyticsVidhya%20Article%20-%20ML%20Model%20approach.ipynb)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "class PreProcessing(BaseEstimator, TransformerMixin):\n",
    "    \"\"\"Custom Pre-Processing estimator for our use-case\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def transform(self, df):\n",
    "        \"\"\"Regular transform() that is a help for training, validation & testing datasets\n",
    "           (NOTE: The operations performed here are the ones that we did prior to this cell)\n",
    "        \"\"\"\n",
    "        pred_var = ['Gender','Married','Dependents','Education','Self_Employed','ApplicantIncome',\\\n",
    "                    'CoapplicantIncome','LoanAmount','Loan_Amount_Term','Credit_History','Property_Area']\n",
    "        \n",
    "        df = df[pred_var]\n",
    "        \n",
    "        df['Dependents'] = df['Dependents'].fillna(0)\n",
    "        df['Self_Employed'] = df['Self_Employed'].fillna('No')\n",
    "        df['Loan_Amount_Term'] = df['Loan_Amount_Term'].fillna(self.term_mean_)\n",
    "        df['Credit_History'] = df['Credit_History'].fillna(1)\n",
    "        df['Married'] = df['Married'].fillna('No')\n",
    "        df['Gender'] = df['Gender'].fillna('Male')\n",
    "        df['LoanAmount'] = df['LoanAmount'].fillna(self.amt_mean_)\n",
    "        \n",
    "        gender_values = {'Female' : 0, 'Male' : 1} \n",
    "        married_values = {'No' : 0, 'Yes' : 1}\n",
    "        education_values = {'Graduate' : 0, 'Not Graduate' : 1}\n",
    "        employed_values = {'No' : 0, 'Yes' : 1}\n",
    "        property_values = {'Rural' : 0, 'Urban' : 1, 'Semiurban' : 2}\n",
    "        dependent_values = {'3+': 3, '0': 0, '2': 2, '1': 1}\n",
    "        df.replace({'Gender': gender_values, 'Married': married_values, 'Education': education_values, \\\n",
    "                    'Self_Employed': employed_values, 'Property_Area': property_values, \\\n",
    "                    'Dependents': dependent_values}, inplace=True)\n",
    "        \n",
    "        return df.as_matrix()\n",
    "\n",
    "    def fit(self, df, y=None, **fit_params):\n",
    "        \"\"\"Fitting the Training dataset & calculating the required values from train\n",
    "           e.g: We will need the mean of X_train['Loan_Amount_Term'] that will be used in\n",
    "                transformation of X_test\n",
    "        \"\"\"\n",
    "        \n",
    "        self.term_mean_ = df['Loan_Amount_Term'].mean()\n",
    "        self.amt_mean_ = df['LoanAmount'].mean()\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Convert `y_train` & `y_test` to `np.array`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = y_train.replace({'Y':1, 'N':0}).as_matrix()\n",
    "y_test = y_test.replace({'Y':1, 'N':0}).as_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create a `pipeline` to make sure that all the preprocessing steps that we do are just a single `scikit-learn estimator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(PreProcessing(),\n",
    "                    RandomForestClassifier())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('preprocessing', PreProcessing()), ('randomforestclassifier', RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False))])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To search for the best `hyper-parameters` (`degree` for `PolynomialFeatures` & `alpha` for `Ridge`), we'll do a `Grid Search`:\n",
    "\n",
    "- Defining `param_grid`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param_grid = {\"randomforestclassifier__n_estimators\" : [10, 20, 30],\n",
    "             \"randomforestclassifier__max_depth\" : [None, 6, 8, 10],\n",
    "             \"randomforestclassifier__max_leaf_nodes\": [None, 5, 10, 20], \n",
    "             \"randomforestclassifier__min_impurity_split\": [0.1, 0.2, 0.3]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Running the `Grid Search`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grid = GridSearchCV(pipe, param_grid=param_grid, cv=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Fitting the training data on the `pipeline estimator`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('preprocessing', PreProcessing()), ('randomforestclassifier', RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impu..._jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False))]),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'randomforestclassifier__n_estimators': [10, 20, 30], 'randomforestclassifier__max_leaf_nodes': [None, 5, 10, 20], 'randomforestclassifier__min_impurity_split': [0.1, 0.2, 0.3], 'randomforestclassifier__max_depth': [None, 6, 8, 10]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's see what parameter did the Grid Search select:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'randomforestclassifier__n_estimators': 30, 'randomforestclassifier__max_leaf_nodes': 20, 'randomforestclassifier__min_impurity_split': 0.2, 'randomforestclassifier__max_depth': 8}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best parameters: {}\".format(grid.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set score: 0.79\n"
     ]
    }
   ],
   "source": [
    "print(\"Validation set score: {:.2f}\".format(grid.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Load the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_df = pd.read_csv('../data/test.csv', encoding=\"utf-8-sig\")\n",
    "test_df = test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Loan_ID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Married</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>Education</th>\n",
       "      <th>Self_Employed</th>\n",
       "      <th>ApplicantIncome</th>\n",
       "      <th>CoapplicantIncome</th>\n",
       "      <th>LoanAmount</th>\n",
       "      <th>Loan_Amount_Term</th>\n",
       "      <th>Credit_History</th>\n",
       "      <th>Property_Area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LP001015</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>No</td>\n",
       "      <td>5720</td>\n",
       "      <td>0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LP001022</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>No</td>\n",
       "      <td>3076</td>\n",
       "      <td>1500</td>\n",
       "      <td>126.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>LP001031</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>2</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>No</td>\n",
       "      <td>5000</td>\n",
       "      <td>1800</td>\n",
       "      <td>208.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LP001035</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>2</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>No</td>\n",
       "      <td>2340</td>\n",
       "      <td>2546</td>\n",
       "      <td>100.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LP001051</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>0</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>No</td>\n",
       "      <td>3276</td>\n",
       "      <td>0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Loan_ID Gender Married Dependents     Education Self_Employed  \\\n",
       "0  LP001015   Male     Yes          0      Graduate            No   \n",
       "1  LP001022   Male     Yes          1      Graduate            No   \n",
       "2  LP001031   Male     Yes          2      Graduate            No   \n",
       "3  LP001035   Male     Yes          2      Graduate            No   \n",
       "4  LP001051   Male      No          0  Not Graduate            No   \n",
       "\n",
       "   ApplicantIncome  CoapplicantIncome  LoanAmount  Loan_Amount_Term  \\\n",
       "0             5720                  0       110.0             360.0   \n",
       "1             3076               1500       126.0             360.0   \n",
       "2             5000               1800       208.0             360.0   \n",
       "3             2340               2546       100.0             360.0   \n",
       "4             3276                  0        78.0             360.0   \n",
       "\n",
       "   Credit_History Property_Area  \n",
       "0             1.0         Urban  \n",
       "1             1.0         Urban  \n",
       "2             1.0         Urban  \n",
       "3             NaN         Urban  \n",
       "4             1.0         Urban  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.predict(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our `pipeline` is looking pretty swell & fairly decent to go the most important step of the tutorial: __Serialize the Machine Learning Model__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Saving Machine Learning Model : Serialization & Deserialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">In computer science, in the context of data storage, serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and reconstructed later in the same or another computer environment.\n",
    "\n",
    "In Python, `pickling` is a standard way to store objects and retrieve them as their original state. To give a simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "list_to_pickle = [1, 'here', 123, 'walker']\n",
    "\n",
    "#Pickling the list\n",
    "import pickle\n",
    "\n",
    "list_pickle = pickle.dumps(list_to_pickle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x03]q\\x00(K\\x01X\\x04\\x00\\x00\\x00hereq\\x01K{X\\x06\\x00\\x00\\x00walkerq\\x02e.'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we load the pickle back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loaded_pickle = pickle.loads(list_pickle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 'here', 123, 'walker']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the `pickled object` to a file as well and use it. This method is similar to creating `.rda` files for folks who are familiar with `R Programming`. \n",
    "\n",
    "__NOTE:__ Some people also argue against using `pickle` for serialization[(1)](#no1). `h5py` could also be an alternative.\n",
    "\n",
    "We have a custom `Class` that we need to import while running our training, hence we'll be using `dill` module to packup the `estimator Class` with our `grid object`.\n",
    "\n",
    "It is advisable to create a separate `training.py` file that contains all the code for training the model ([See here for example](https://github.com/pratos/flask_api/blob/master/flask_api/utils.py)).\n",
    "\n",
    "- To install `dill`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: dill in /home/pratos/miniconda3/envs/ordermanagement/lib/python3.5/site-packages\r\n"
     ]
    }
   ],
   "source": [
    "!pip install dill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import dill as pickle\n",
    "filename = 'model_v1.pk'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('../flask_api/models/'+filename, 'wb') as file:\n",
    "\tpickle.dump(grid, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our model will be saved in the location above. Now that the model `pickled`, creating a `Flask` wrapper around it would be the next step.\n",
    "\n",
    "Before that, to be sure that our `pickled` file works fine -- let's load it back and do a prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('../flask_api/models/'+filename ,'rb') as f:\n",
    "    loaded_model = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_model.predict(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, we already have the `preprocessing` steps required for the new incoming data present as a part of the `pipeline` we just have to run `predict()`. While working with `scikit-learn`, it is always easy to work with `pipelines`. \n",
    "\n",
    "`Estimators` and `pipelines` save you time and headache, even if the initial implementation seems to be ridiculous. Stich in time, saves nine!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Creating an API using Flask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We'll keep the folder structure as simple as possible:\n",
    "\n",
    "![Folder Struct](https://raw.githubusercontent.com/pratos/flask_api/master/notebooks/images/flaskapp3.png)\n",
    "\n",
    "There are three important parts in constructing our wrapper function, `apicall()`:\n",
    "\n",
    "- Getting the `request` data (for which predictions are to be made)\n",
    "\n",
    "- Loading our `pickled estimator`\n",
    "\n",
    "- `jsonify` our predictions and send the response back with `status code: 200`\n",
    "\n",
    "HTTP messages are made of a header and a body. As a standard, majority of the body content sent across are in `json` format. We'll be sending (`POST url-endpoint/`) the incoming data as batch to get predictions.\n",
    "\n",
    "(__NOTE:__ You can send plain text, XML, csv or image directly but for the sake of interchangeability of the format, it is advisable to use `json`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "\"\"\"Filename: server.py\n",
    "\"\"\"\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.externals import joblib\n",
    "from flask import Flask, jsonify, request\n",
    "\n",
    "app = Flask(__name__)\n",
    "\n",
    "@app.route('/predict', methods=['POST'])\n",
    "def apicall():\n",
    "\t\"\"\"API Call\n",
    "\t\n",
    "\tPandas dataframe (sent as a payload) from API Call\n",
    "\t\"\"\"\n",
    "\ttry:\n",
    "\t\ttest_json = request.get_json()\n",
    "\t\ttest = pd.read_json(test_json, orient='records')\n",
    "\n",
    "\t\t#To resolve the issue of TypeError: Cannot compare types 'ndarray(dtype=int64)' and 'str'\n",
    "\t\ttest['Dependents'] = [str(x) for x in list(test['Dependents'])]\n",
    "\n",
    "\t\t#Getting the Loan_IDs separated out\n",
    "\t\tloan_ids = test['Loan_ID']\n",
    "\n",
    "\texcept Exception as e:\n",
    "\t\traise e\n",
    "\t\n",
    "\tclf = 'model_v1.pk'\n",
    "\t\n",
    "\tif test.empty:\n",
    "\t\treturn(bad_request())\n",
    "\telse:\n",
    "\t\t#Load the saved model\n",
    "\t\tprint(\"Loading the model...\")\n",
    "\t\tloaded_model = None\n",
    "\t\twith open('./models/'+clf,'rb') as f:\n",
    "\t\t\tloaded_model = pickle.load(f)\n",
    "\n",
    "\t\tprint(\"The model has been loaded...doing predictions now...\")\n",
    "\t\tpredictions = loaded_model.predict(test)\n",
    "\t\t\n",
    "\t\t\"\"\"Add the predictions as Series to a new pandas dataframe\n",
    "\t\t\t\t\t\t\t\tOR\n",
    "\t\t   Depending on the use-case, the entire test data appended with the new files\n",
    "\t\t\"\"\"\n",
    "\t\tprediction_series = list(pd.Series(predictions))\n",
    "\n",
    "\t\tfinal_predictions = pd.DataFrame(list(zip(loan_ids, prediction_series)))\n",
    "\t\t\n",
    "\t\t\"\"\"We can be as creative in sending the responses.\n",
    "\t\t   But we need to send the response codes as well.\n",
    "\t\t\"\"\"\n",
    "\t\tresponses = jsonify(predictions=final_predictions.to_json(orient=\"records\"))\n",
    "\t\tresponses.status_code = 200\n",
    "\n",
    "\t\treturn (responses)\n",
    "\n",
    "```\n",
    "\n",
    "Once done, run: `gunicorn --bind 0.0.0.0:8000 server:app`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate some prediction data and query the API running locally at `https:0.0.0.0:8000/predict`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"Setting the headers to send and accept json responses\n",
    "\"\"\"\n",
    "header = {'Content-Type': 'application/json', \\\n",
    "                  'Accept': 'application/json'}\n",
    "\n",
    "\"\"\"Reading test batch\n",
    "\"\"\"\n",
    "df = pd.read_csv('../data/test.csv', encoding=\"utf-8-sig\")\n",
    "df = df.head()\n",
    "\n",
    "\"\"\"Converting Pandas Dataframe to json\n",
    "\"\"\"\n",
    "data = df.to_json(orient='records')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"Loan_ID\":\"LP001015\",\"Gender\":\"Male\",\"Married\":\"Yes\",\"Dependents\":\"0\",\"Education\":\"Graduate\",\"Self_Employed\":\"No\",\"ApplicantIncome\":5720,\"CoapplicantIncome\":0,\"LoanAmount\":110.0,\"Loan_Amount_Term\":360.0,\"Credit_History\":1.0,\"Property_Area\":\"Urban\"},{\"Loan_ID\":\"LP001022\",\"Gender\":\"Male\",\"Married\":\"Yes\",\"Dependents\":\"1\",\"Education\":\"Graduate\",\"Self_Employed\":\"No\",\"ApplicantIncome\":3076,\"CoapplicantIncome\":1500,\"LoanAmount\":126.0,\"Loan_Amount_Term\":360.0,\"Credit_History\":1.0,\"Property_Area\":\"Urban\"},{\"Loan_ID\":\"LP001031\",\"Gender\":\"Male\",\"Married\":\"Yes\",\"Dependents\":\"2\",\"Education\":\"Graduate\",\"Self_Employed\":\"No\",\"ApplicantIncome\":5000,\"CoapplicantIncome\":1800,\"LoanAmount\":208.0,\"Loan_Amount_Term\":360.0,\"Credit_History\":1.0,\"Property_Area\":\"Urban\"},{\"Loan_ID\":\"LP001035\",\"Gender\":\"Male\",\"Married\":\"Yes\",\"Dependents\":\"2\",\"Education\":\"Graduate\",\"Self_Employed\":\"No\",\"ApplicantIncome\":2340,\"CoapplicantIncome\":2546,\"LoanAmount\":100.0,\"Loan_Amount_Term\":360.0,\"Credit_History\":null,\"Property_Area\":\"Urban\"},{\"Loan_ID\":\"LP001051\",\"Gender\":\"Male\",\"Married\":\"No\",\"Dependents\":\"0\",\"Education\":\"Not Graduate\",\"Self_Employed\":\"No\",\"ApplicantIncome\":3276,\"CoapplicantIncome\":0,\"LoanAmount\":78.0,\"Loan_Amount_Term\":360.0,\"Credit_History\":1.0,\"Property_Area\":\"Urban\"}]'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"POST <url>/predict\n",
    "\"\"\"\n",
    "resp = requests.post(\"http://0.0.0.0:8000/predict\", \\\n",
    "                    data = json.dumps(data),\\\n",
    "                    headers= header)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': '[{\"0\":\"LP001015\",\"1\":1},{\"0\":\"LP001022\",\"1\":1},{\"0\":\"LP001031\",\"1\":1},{\"0\":\"LP001035\",\"1\":1},{\"0\":\"LP001051\",\"1\":1}]'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"The final response we get is as follows:\n",
    "\"\"\"\n",
    "resp.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We have half the battle won here, with a working API that serves predictions in a way where we take one step towards integrating our ML solutions right into our products. This is a very basic API that will help with proto-typing a data product, to make it as fully functional, production ready API a few more additions are required that aren't in the scope of Machine Learning. \n",
    "\n",
    "There are a few things to keep in mind when adopting API-first approach:\n",
    "\n",
    "- Creating APIs out of sphagetti code is next to impossible, so approach your Machine Learning workflow as if you need to create a clean, usable API as a deliverable. Will save you a lot of effort to jump hoops later.\n",
    "\n",
    "- Try to use version control for models and the API code, `Flask` doesn't provide great support for version control. Saving and keeping track of ML Models is difficult, find out the least messy way that suits you. [This article](https://medium.com/towards-data-science/how-to-version-control-your-machine-learning-task-cad74dce44c4) talks about ways to do it.\n",
    "\n",
    "- Specific to `sklearn models` (as done in this article), if you are using custom `estimators` for preprocessing or any other related task make sure you keep the `estimator` and `training code` together so that the model pickled would have the `estimator` class tagged along. \n",
    "\n",
    "Next logical step would be creating a workflow to deploy such APIs out on a small VM. There are various ways to do it and we'll be looking into those in the next article.\n",
    "\n",
    "Code & Notebooks for this article: [pratos/flask_api](https://github.com/pratos/flask_api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Sources & Links:__\n",
    "\n",
    "[1]. <a id='no1' href=\"http://www.benfrederickson.com/dont-pickle-your-data/\">Don't Pickle your data.</a>\n",
    "\n",
    "[2]. <a id='no2' href=\"http://www.dreisbach.us/articles/building-scikit-learn-compatible-transformers/\">Building Scikit Learn compatible transformers.</a>\n",
    "\n",
    "[3]. <a id='no2' href=\"http://flask.pocoo.org/docs/0.10/security/#json-security\">Using jsonify in Flask.</a>\n",
    "\n",
    "[4]. <a id='no2' href=\"http://blog.luisrei.com/articles/flaskrest.html\">Flask-QuickStart.</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
